Skip to content

Support TiDB #3001

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 7 commits into
base: master
Choose a base branch
from
Draft

Support TiDB #3001

wants to merge 7 commits into from

Conversation

Torch3333
Copy link
Contributor

@Torch3333 Torch3333 commented Sep 4, 2025 •
edited
Loading

Description

This adds support for TiDB versions 6.5, 7.5, and 8.5, a fully MySQL-compliant database. Connection to TiDB is done through the MySQL Connector/J JDBC driver.

TiDB supports the REPEATABLE_READ and READ_COMMITED transaction isolation level. Since the ScalarDB transaction API sets a SERIALIZABLE isolation level by default not supported by TiDB, users must overwrite it by adding scalar.db.jdbc.isolation_level=REPEATABLE_READ or scalar.db.jdbc.isolation_level=READ_COMMITED to their configuration to use ScalarDB transactional API interfaces.

A sample ScalarDB configuration to use TiDB is as follows.

scalar.db.storage=jdbc
scalar.db.contact_points=jdbc:mysql://localhost:4000/
scalar.db.username=root
scalar.db.password=tidb
# Required if using the transactional API
scalar.db.jdbc.isolation_level=REPEATABLE_READ

Related issues and/or PRs

N/A

Changes made

  • Adds TiDB integration tests to the CI

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

  • I have commented my code, particularly in hard-to-understand areas.
  • I have updated the documentation to reflect the changes.
  • I have considered whether similar issues could occur in other products, components, or modules if this PR is for bug fixes.
  • Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
  • Tests (unit, integration, etc.) have been added for the changes.
  • My changes generate no new warnings.
  • Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

N/A

Release notes

TiDB versions 6.5, 7.5, and 8.5 are now supported as underlying storage by using MySQL Connector/J JDBC driver.

@Torch3333 Torch3333 self-assigned this Sep 4, 2025
@Torch3333 Torch3333 added the enhancement New feature or request label Oct 3, 2025
@Torch3333 Torch3333 marked this pull request as ready for review October 3, 2025 08:05
@Torch3333 Torch3333 requested review from a team, brfrn169, Copilot, feeblefakie and komamitsu and removed request for a team October 3, 2025 08:05
Copilot
Copilot AI reviewed Oct 3, 2025
View reviewed changes
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds TiDB support to ScalarDB as an underlying storage by enabling connection through MySQL Connector/J JDBC driver. It supports TiDB versions 6.5, 7.5, and 8.5, with configuration to override the default SERIALIZABLE isolation level to REPEATABLE_READ or READ_COMMITTED since TiDB doesn't support SERIALIZABLE.

  • Added isolation level configuration support in JDBC environment setup
  • Implemented TiDB-specific handling for unsupported data types in import tests
  • Added comprehensive CI integration tests for TiDB versions 6.5, 7.5, and 8.5

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
core/src/integration-test/java/com/scalar/db/storage/jdbc/JdbcEnv.java Adds isolation level configuration property support for test environment
core/src/integration-test/java/com/scalar/db/storage/jdbc/JdbcAdminImportTestUtils.java Implements TiDB detection and specific data type filtering for import tests
.github/workflows/ci.yaml Adds CI integration test jobs for TiDB versions 6.5, 7.5, and 8.5

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

komamitsu
komamitsu approved these changes Oct 6, 2025
View reviewed changes
Copy link
Contributor

@komamitsu komamitsu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 👍

@brfrn169
Copy link
Collaborator

brfrn169 commented Oct 7, 2025

Since the ScalarDB transaction API sets a SERIALIZABLE isolation level by default not supported by TiDB

@Torch3333 I don’t think we set the SERIALIZABLE isolation level by default. We only set the isolation level when users specify the scalar.db.jdbc.isolation_level property. What do you think?

@feeblefakie
Copy link
Contributor

@Torch3333 What would happen if users don't specify scalardb.jdbc.isolation_level=REPEATABLE_READ or READ-COMMITTED?

BTW, TiDB seems to be linearizable, independent of isolation levels. So, any isolation levels can probaly be fine for correctness. (cc: @brfrn169 )
https://docs.pingcap.com/tidb/stable/system-variables/#tidb_guarantee_linearizability-new-in-v50

@Torch3333
Merge remote-tracking branch 'origin/master' into tidb_support
3ae5afe 
# Conflicts:
#	.github/workflows/ci.yaml
@Torch3333
Copy link
Contributor Author

Torch3333 commented Oct 20, 2025 •
edited
Loading

@brfrn169 @feeblefakie @komamitsu

@Torch3333 I don’t think we set the SERIALIZABLE isolation level by default. We only set the isolation level when users specify the scalar.db.jdbc.isolation_level property. What do you think?

You are right. I mistakenly thought ScalarDB was setting the JDBC transaction isolation level for all the transaction managers, but in fact, it is only set for the JDBC transaction manager, not for Consensus Commit.
As discussed with @brfrn169 and @feeblefakie, we decided that setting it for the JDBC transaction manager was not required. From now on, the storage default isolation level will be used. For better clarity for our users, I made these changes in a separate PR #3076

What would happen if users don't specify scalardb.jdbc.isolation_level=REPEATABLE_READ or READ-COMMITTED?

TiDB uses the REPEATABLE-READ as the default level. READ-COMMITED is also an accepted value. Setting SERIALIZABLE will cause the driver to throw this error. The documentation explains it here.

The isolation level 'SERIALIZABLE' is not supported. Set tidb_skip_isolation_level_check=1 to skip this error

Other than that, I finally found a document listing the features of the MySQL Connector/J driver that TiDB does not support. As far as my understanding goes, I don't think ScalarDB uses any of the incompatibilities, so this confirms that using this driver instead of the TiDB modified version of Connector/J is fine.

@Torch3333 Torch3333 mentioned this pull request Oct 20, 2025
Merged
7 tasks
@Torch3333
Copy link
Contributor Author

I will put this PR in draft until this blocking dependent PR is merged.
#3076

@Torch3333 Torch3333 marked this pull request as draft October 20, 2025 06:23
@Torch3333
Copy link
Contributor Author

@komamitsu You raised a point during the demo meeting more than 2 weeks ago about how the driver handles load balancing. There seems to be a library to handle just that, but I will investigate it once other higher-priority tasks are completed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@komamitsu komamitsu komamitsu approved these changes

@brfrn169 brfrn169 Awaiting requested review from brfrn169 brfrn169 was automatically assigned from scalar-labs/scalardb

@feeblefakie feeblefakie Awaiting requested review from feeblefakie feeblefakie was automatically assigned from scalar-labs/scalardb

At least 2 approving reviews are required to merge this pull request.

Assignees

@Torch3333 Torch3333

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Torch3333 @brfrn169 @feeblefakie @komamitsu